Repository navigation
sync: batch small files, skip unchanged manifests, probe presence on its own - #26
Open
ChakraFusion wants to merge 4 commits into
Open
ChakraFusion wants to merge 4 commits into
ChakraFusion wants to merge 4 commits into
Conversation
|
@ChakraFusion is attempting to deploy a commit to the sivadaboi's projects Team on Vercel. A member of the Team first needs to authorize it. |
This was referenced Oct 4, 2026
ChakraFusion
force-pushed
the
pr/sync-speed
branch
from
October 13, 2026 17:55
cf9269a to
14c12ba
Compare
ChakraFusion
force-pushed
the
pr/sync-speed
branch
from
October 15, 2026 12:56
14c12ba to
e6b2156
Compare
Author
|
Rebased onto today's |
…its own Performance - Batched pulls (protocol revision 2): small files are fetched up to 256 per request (4MB), 4 requests in flight, over a new /api/p2p/files route. Each file is still written through the PatchWriter (verified, fsynced, renamed). One request per file cost ~200ms on a real LAN; on loopback 5000 files went from 42s to 12s. Peers below revision 2 and relay connections keep the per-file path. - Unchanged games: when this device still holds the agreed base, the manifest request carries ifHash and a peer that still holds it too answers "unchanged" instead of sending the manifest. Older peers ignore the parameter; their full answer is used, not fetched twice. - Manifests are gzipped for clients that accept it (every Go client does, so older versions read them too): ~5.6x smaller with real hashes. - Manifests and batches get a 10-minute client timeout instead of 30s; a large manifest over a VPN never arrived inside 30s, so those games never synced. - An in-sync pass also mirrors the peer's latest snapshot, which only a pull did before; games identical everywhere from the start showed history on one device only. - A batch is bounded by what the responder counts (each requested block at the file's full block size, at most the file's real size), so a batch of many small files with a few larger ones is never refused as too large; a batch refused anyway is fetched again as two halves. Robustness - Presence: paired LAN peers are probed every 10s on their own ticker. Probes used to ride along with the minute-long reconcile, which waits for a whole sync-all pass; peers reached over a VPN (no UDP broadcast) showed online/offline minutes late. Probe rounds are serialized. - Games whose save folder is not on this device are skipped by sync-all and dropped from the retry queue instead of failing every 20s. - "syncing X with Y" is logged only when there is something to do; most passes produced only "already in sync" lines, which rotated everything else out of the log within hours. - The interrupted-pull safety test also runs over batched pulls. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every peer started each run with whatever status the database held, and a peer counts as offline only after three failed probes. For a PC switched off overnight that meant being shown online after every start until three probes - the first ten seconds after the background loops started - had failed. - The presence loop probes at once on start, then every 10s. - The three-miss tolerance is for devices heard from during this run (answered a probe, or seen by discovery, a request or the relay since start). A peer with no sign of life since start goes offline on its first unanswered probe. Adds an opt-in e2e measurement (OPENSAVE_PRESENCE_TIMING) of how long a stopped device is shown online: 30s, by design, for one seen this run.
… time The patch writer settles a pulled file as OpenSave's own when it commits it (owntouch, Liquid-co#23). The batched pull then gave the file the peer's modification time without marking it again, as the per-file path does, so the exact time-and-size comparison failed and the watcher took the pull for the game saving: an automatic snapshot after every sync of small files, and a sync back of a change this device never made. Found once Liquid-co#52's test ran against this branch on dev-public (TestLauncherFile_SteamRewritingRemotecacheIsNoSaveChange: 4/4 failures before, 10/10 passes after, in separate runs). Both batched paths now set the time the way the per-file path does (setPulledMtime: mark, set, settle). Test: TestSetPulledMtime_KeepsThePullOpenSavesOwn fails without the marking. Co-Authored-By: Claude Opus 5.5 <[email protected]>
ChakraFusion
force-pushed
the
pr/sync-speed
branch
from
October 17, 2026 17:42
e6b2156 to
13cbf65
Compare
…kept yet An in-sync pass shares the peer's latest snapshot so a game identical everywhere from the start (Steam Cloud, a copied folder) has history on every device. It mirrored it whenever the peer's latest snapshot was newer, also on a device that already keeps a snapshot of its save as it stands: a second copy of the same save, named as synced from a peer that sent nothing. Even the peer's first snapshot of an empty folder was taken this way. Found by TestLauncherFile_SteamRewritingRemotecacheIsNoSaveChange, which failed or passed depending on which device tracked the game a few milliseconds later (always on Linux and Windows CI for Liquid-co#27). The mirror is now taken only where this device keeps no snapshot of the current save yet: the content recorded with its last snapshot (the watcher's baseline) is not the content now (keepsCurrentSave). Same in the quick "unchanged" path. Test: TestSync_InSyncDoesNotMirrorASaveAlreadyKept fails without the check; TestSync_InSyncMirrorsPeerSnapshot still passes. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Author
|
Rebased onto
The full unit and e2e suites pass locally on the rebased branch. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #21.
Based on the "Keep mine" PR (#22): its first commit is that PR. Only the last two commits are new here. I'll rebase once that PR is merged.
Why
A save of many small files (Project Zomboid: ~240k map chunks) cost one HTTP request per file. That is ~200 ms each on a real LAN, so a pull of 50 MB took hours. On top of that, every periodic pass fetched and compared the full manifest for every game, even when nothing had changed.
What
Batched pulls (protocol revision 2)
/api/p2p/filesroute.PatchWriter(verified, fsynced, renamed).Unchanged games
ifHashand the peer answers "unchanged" instead of sending the manifest.Gzip and timeouts
Smaller items
Presence (second commit)
Compatibility
Mixed versions work in both directions:
Accept-Encoding.Tests
batch_test.go: small files are batched, older peers are not, a batch stays under what the responder counts, and a refused batch is split.quick_test.go: the unchanged shortcut, real changes still sync, and an older peer's manifest is not fetched twice.manifest_gzip_test.go,presence_test.go,missing_skip_test.go.partial_safety_test.go: an interrupted batched pull never turns into deletions.e2e/batch_bench_test.go: opt-in measurement.🤖 Generated with Claude Code